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EXECUTIVE SUMMARY 


wn 


The Science of Learning and Readiness (SoLaR) project seeks to demonstrate to Defense and other 
Government stakeholders the “art of the possible” for high-quality distributed learning and to create a practical 
guide for how to infuse such qualities into the broader Department of Defense (DoD) distributed learning 
ecosystem. 


Description of Report 


This report seeks to provide an understanding of a State-of-the-Art Learning Organization. The features of an 
exemplar learning organization at scale were identified based on a broad review of the literature. The 
summarized key findings are below 


1) Principles of human learning from the learning sciences hold within learning at scale environments. 

2) Human learning within these environments must be supported by technology. 

3) The technology must provide data on the learning process to the organization. 

4) Learning organizations must use data to support both the learners in the form of learning, social, and 
academic guidance, and the members of the organization to provide training, support, and 
recognition. 


Within this report, we gathered real-world examples from exemplar organizations and provide an example of a 
growth path for an exemplar organization. We identified highly influential learning organizations from 
industry or post-secondary settings that are recognized for both their quality education and ability to operate at 
scale within adult distributed learning environments. Examples of exemplar practices in action were identified 
from interview data spanning academia, non-profit, and industry. From this data, the report describes 
important factors the organizations used to scale up, identifies generalizable principles, and highlights the 
characteristics that make each organization uniquely effective. The report concludes with an example of an 
ideal scaled-up process. This example provides a pathway based on best practices for learning organizations to 
scale up. 


Conclusions 


This report presents examples of how a learning organization can scale up its training and education to meet 
the needs of its learners. It is possible to scale up a learning or training organization, but it takes time, planning 
and effort across the organization. Our interviews revealed that experts suggest a “think-before-you-act” 
mentality when scaling up, as it is essential to consider scale from the start of an initiative. Otherwise, 
opportunity can be overlooked because of a single-minded focus on fulfilling the immediate cohort’s needs. 
Every interviewee mentioned having a responsive leadership is part of the scaling up. All interviewees kept 
reiterating the importance and effectiveness of data in student success and staff training. This included both 
automated data from the systems such as that provided by xAPI and data on the use and abilities of 
participants such as would be collected by usability and experimental data collection. A final theme was that 
along with the development of the technology infrastructure, supporting comprehensive human infrastructure 
is required. This requires a full spectrum of support for students and instructors as well as teams of specialists 
for developing and maintaining the learning ecosystems. The report concludes with a concrete example of our 
identified principles for scaling up using a fictional company. This fictional exemplar provides examples of the 
state-of-the-art for learning at scale integrated into a scenario to model the transition process of becoming a 
learning organization at scale. 


EXEMPLAR REPORT: 
A PATH TOWARD LEARNING AT SCALE 


This report explores the processes of how best to scale up learning organizations. We start by providing a 
summary of interviews with individuals at learning organizations that have successfully scaled up, and those 
individuals provide details about the lessons learned from the scale-up process. We conclude the report by 
presenting a fictional example of how an organization could scale up based on principles established from a 
review of the research literature and our primary data collection. 


1.0 SCALING UP TO MEET DEMANDS FOR EDUCATION AND READINESS 


Learning organizations are rapidly adapting how they provide education and training. These changes are both 
technology driven and practical to provide education and training to greater numbers of learners at a rapid 
pace (Graesser, Hu, & Ritter, 2019). Many of these learners are immersed in online learning environments. 
For example, there are estimated to be 6,651,536 students enrolled in online education courses at the 
postsecondary level (National Center for Education Statistics Fast Facts, 2018). This number accounts for 
almost 34% of the current student population (National Center for Education Statistics, 2018). This trend of 
growth has been constant in the United States (Allen & Seaman, 2016). 


The high pressure of providing education and training within this rapidly growing technological environment 
often requires rapid decisions based on limited information. This has led well-meaning decision makers to 
adopt fallacies in their decision making. There is a tendency to cling to traditional methods such as in-person 
lectures instead of innovating, because of the belief that eLearning and flipped/technology-enhanced 
classrooms are less effective than traditional classrooms. However, this is not the case. eLearning (Means, 
Toyama, Murphy, & Baki, 2013) can be as effective as traditional classrooms and in some cases more effective 
with proper supports, such as cooperative learning spaces, interactive multimedia, and virtual environments 
(Davis, Chen, Hauff, & Houben, 2018). For this transition to happen, there needs to be deliberate 
consideration of the needs of the learner and the organization, support for those needs, and willingness to 
explore state-of-the-art techniques to address the needs. 


1.1 WHAT IS LEARNING AT SCALE 


Learning at scale is defined as “the technologies, pedagogies, analyses, and theories of learning and teaching 
that take place with a large number of learners and a high ratio of learners to facilitators” (Roll, Russel, & 
GaSevic, 2018, p. 473). The number of students seeking learning via online formats has continuously grown 
over the last few years. Massively Open Online Courses (MOOCs), the classic example of learning at scale, 
have grown to more than 4000 active course (Davis et al., 2018). The other form of ‘at scale’ involves offering 
many smaller sections of the same course that total thousands of students served. 


1.2 SUMMARY OF BEST PRACTICES 


In previous work, we conducted a broad literature review and survey of learning organizations to identify the 
current state of the art for learning at scale. Scaled-up learning environments are similar to other online 
learning environments with the exception that scale limits the amount of potential human interaction within 
the environment. In brief, the four high level key findings from the review are: 


1) The basic principles of human learning from the learning sciences still hold within blended and 
learning at scale environments. 

2) Human learning within these environments must be supported by technology. 

3) The technology must provide data on the learning process to the learning organization. 

4) Learning organizations must use data to support both the learners in the form of learning, social, and 
academic guidance, and the members of the learning institutions to provide training, support, and 
recognition. 


A follow-up survey of public, private, and academic learning organizations found a disconnect between higher 
ratings of the importance and lower incorporation of the identified best practices. This disconnect indicates a 
concerning divide but also a potential readiness within learning organizations for transitioning to the best 
practices. 


The observed discrepancy sets forth the need to understand how these best practices could be implemented. 
The current report targets this need in two ways. First, we conducted a series of interviews with exemplar 
organizations that are working on learning at scale. These interviews provide targeted exemplar practices and 
lessons learned from the process of scaling up. Second, we provide an example of what a learning organization 
could look like as it scales up using some of the best practices. 


2.0 WHAT DOES AN EXEMPLAR LOOK LIKE IN PRACTICE? 


No single example of a scaled-up learning organization is perfect. However, there are organizations that are 
aspiring towards it. To collect examples, we turn to successful learning organizations to understand their scale 
up process. The best practices of these exemplar organizations were collected using a semi-structured interview 
methodology. 


2.1 INTERVIEW DATA 


2.1.1 OVERVIEW 


In this section, empirical methods were used to identify the major factors for organizational scale-up. We 
conducted semi-structured interviews with interviewees in different organizations. The goal for these 
interviews was to conduct a case study to collect best practices for designing, delivering, and managing 
distributed learning at scale. This section is organized as follows: Section 3.1.2 (i.e., Methodology) includes a 
brief summary of the semi-structured interview method, wherein we present the nature of a semi-structured 
interview and why we chose to use it in our investigation. Then we describe the procedure used to conduct 
these interviews, including our interview protocol. Section 3.2 consists of the description and the exemplar 
examples of four organizations in the following order: academia, non-profit, and industry. 


2.1.2 METHODOLOGY 


RESEARCH DESIGN 


Semi-structured or open-ended interviews are formatted to allow the interviewer to ask several standard or 
base questions of each participant, while still retaining the option to ask follow-up questions, depending on the 
respondent’s answers (Mertler, 2019). The advantages of semi-structured interviews are that they provide a 
richness and perspective in data sets that cannot be garnered from structured interviews, which are limited to 
predetermined questions. Also, semi-structured interviews allow for a more natural conversation flow that 
encourages the respondent to elaborate at more instinctive times throughout the interview process. In the 
current research, semi-structured interviews were used. The purpose of the interviews was to collect data 
regarding the best practices for designing, delivering, and managing distributed learning at scale in academia, 
non-profit settings, and industry. The interview data provides opportunities for generating evidence to engage 
stakeholders in a current and relevant dialogue regarding distributed learning at scale. 


INTERVIEW SELECTION 


For our interview process, we searched for examples of successful scaling up for learning within public, 
private, and academic sectors. Our interviewees were recruited based on evidence of effectiveness and 
convenience of having participants agree to interviews. This process consisted of interviews from Arizona 
State University’s ASU Online (EdPlus) for the academic sector, Squirrel AI Learning and Kaplan for the 
private sector. Because our interviewee for Kaplan had switched positions, we were also able to get 
information on the Chan Zuckerberg Initiative (CZI) in the non-profit sector. Detailed information on the 
qualifications of the organizations as exemplars are provided before the summaries of each interview. 


INTERVIEW DESCRIPTION 


The interview itself consisted of four sections. The questions in each section were designed to elicit 
information on the organization, the current learning ecosystem and scale-up processes, application of 
technology for scaling up, and experiences of how to scale up. Brief descriptions of the four categories are 
provided below. The interview protocol is provided in Appendix A. 


Section 1: Overview and Understanding of the Organizations. Initial interview questions elicited 
information about the respondents’ background and the background of their organization. Responses 
provided a general idea about the organization’s size, objectives, and online learning platforms. 


Section 2: Current Ecosystems and Scale Up. These questions focused on exploring the practices and 
experiences of the respondents during scaling of learning in their organizations. Specifically, the practices 
explored include organizational changes, learning support, technology support, instructional strategies, 
organizational communication, the support structures within the organization, and multidisciplinary 
collaboration. 


Section 3: The Application of Technology. The purpose of this section was to gain insight into how the 
organizations adopted technologies in all aspects of scaling up, such as internal communication, staff 
training, and instructor and student evaluation methods. These questions uncovered information about the 
role of different learning technologies in the process of scaling learning. 


Section 4: Prior Experiences, Lessons Learned, and Future Goals. Interviews were concluded by asking 
participants to reflect on previous experiences and identify the important lessons they learned. Also, 
respondents were asked to outline the five-year goals of their organization’s future. This final set of 
questions sought to identify foundational considerations for eLearning organizations as new technologies 
and new learning science emerges. 


PROCEDURE 


Participants were selected and agreed to be interviewed. The interviews were conducted utilizing a semi- 
structured format containing nine open-ended questions. The questions were designed to collect case studies 
from successful eLearning implementation in the institutional arenas mentioned previously (academia, non- 
profit, and industry) for the purposes of providing a sketch of current practice in distributed learning and 
expanding the conversation outward regarding best practices in distributed learning to relevant stakeholders 
engaged in distributed learning at scale. 


Currently, eLearning is designed based on multiple learning theories that seek to explain and explore how 
students learn effectively. From our literature search, we discovered that instructional designs are developed 
with several guiding principles. Our initial findings suggested that 1) learning 1s as effective in distance modes 
of delivery as in traditional face-to-face classes; 2) blended learning scenarios (by blended learning, we stipulate 
some combination of face-to-face instruction and distance instruction) may be more effective than traditional 
face-to-face or purely distance learning paradigms; 3) instructors and students are unaware of the equality of 
distance education and traditional instruction; 4) students experience more efficient learning as they move into 
more self-paced and self-directed study; 5) competency-based learning benefits students by removing time from 
the forefront of learning and, instead, elevates competent task performance, thus acknowledging that learning 
can take place outside of formal settings; 6) user experience plays a critical role in student engagement; 6) 
individual learning is diverse, frequent, social, and collaborative; and 7) learners and institutions need a way to 
track their personalized learning pathway to see what and where learning has occurred and to plan for future 
learning experiences. 


The detailed protocol for the interviews is described below. Essentially, the semi-structured interviews were 
performed and recorded. Next, the recordings were transcribed and analyzed. Finally, results in the form of 
emergent themes were organized to summarize findings. The interviews were divided into four question 
groupings which were:1) questions aimed at providing an overview and understanding of the organization; 2) 
questions about the organizations’ application of technology in their educational setting; 3) questions about the 
currently used learning ecosystems across the public and private sectors and how the organization scales-up 
learning; and 4) questions that ask about prior experiences, lessons learned, and future goals. 


CODING INTERVIEW DATA 


To generate evidence about the best practices in academia and industry, interview data were coded deductively 
for identifying emerging themes. In qualitative data coding, this deductive approach is also known as a top- 
down approach. This approach requires that researchers formulate an a priori coding scheme in which the 
codes are developed in advance of examining the interview data. Eligible codes are based on the interview 
questions, prior knowledge, or the theoretical perspectives from a literature review or report. 


For our purposes, four primary codes were established. The codes used were 1) institutional support; 2) 
student support; 3) technology support; and 4) external collaboration. After establishing the coding scheme, we 
applied the codes to analyze the interview text using ATLAS.ti. This initial coding step revealed that the data 
set contained more complex themes and details that needed to be explored. A set of subcodes was developed 
by expanding the four primary codes (See Table 1 in Appendix B). The subcodes under each a priori code 
uncovered several common themes discussed by all participants. To get a holistic picture of best practices 
across different organizations, subcodes were merged into three categories including: 1) technological 
infrastructure; 2) training; and 3) governance. The coding information is organized based on these three 
categories. The specific codes, definitions, and examples are provided in Appendix B Table 1. 
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2.2 INTERVIEW RESULTS 


2.2.1 ACADEMIA: ASU ONLINE (EDPLUS) 


As a division of EdPlus (Arizona State University (ASU) online arm), ASU Online provides degree- 

earning education in a digital format. ASU Online uses the same faculty as the rest of the campus and adheres 
to the university’s charter to “make education available to anyone, anywhere”. The programs at ASU Online 
have grown by leaps and bounds. In only six years (2014-2020), the institution has grown from fewer than 400 
students enrolled in online programs to more than 30,000 students to date. ASU Online offers more than 200 
fully online degrees and has been named both the premier university in the United States for innovation and 
ranks among the top ten universities for quality online education, according to U.S. News & World Report. 


EdPlus is an ASU global initiative that encompasses multiple educational projects for enhancing education, 
including the ASU Online degree programs, a continuing and professional education division, a partnership 
with the Mayo Clinic for providing joint medical education programs, and an earned admissions initiative for 
students suffering from previous academic performance deficits in which students receive access to for-credit, 
cost-effective, first-year college-level courses. Other educational projects include the PLuS Alliance partnership 
for solving global problems with King’s College London and the University of New South Wales Sydney 
(UNSW Sydney), and the Starbucks College Achievement Plan, which is a partnership between ASU Online 
and Starbucks to provide employees access to receive a bachelor’s degree. 


Additionally, EdPlus is dedicated to fostering emerging education through the Education for Humanity 
initiative, which provides educational access and opportunity for refugees and displaced peoples through 
personalized digital learning programs and through the MasterCard Foundation Scholar Community 
Platform-BAOBAB which builds social learning networks for young African scholars. 


Finally, EdPlus encompasses The Action Lab, a teaching and learning laboratory engaged in research in 
digital teaching and learning, whose mission is to provide continuous improvement in online programs and 
student success through quality educational research. Our respondent from academia is on staff at The Action 
Lab as a research analyst whose focus is on associating student success to behavior patterns in online and 
open-scale mathematics courses. 


EXEMPLAR EXAMPLES 


We interviewed Dr. James Cunningham, a senior research analyst at ASU EdPlus. Dr. Cunningham works at 
the Action Lab at EdPlus, which is a dedicated teaching and learning laboratory grounded in cognitive 
science, adaptive learning, social science, and learning sciences to create new methods of digital learning 
research, as well as translate these methods into improving student outcomes at scale. His team plays a key 
role in scaling up by leveraging student-generated data to improve student success as they are working on the 
ASU open scale courses. As an example, Dr. Cunningham observed that because there is a clear distinction 
between the full-time, on-ground students and the part-time, online students in characteristics like time 
management, motivation, and learning goals, etc., an instructor who has taught in both a classroom setting 
and an online setting might not know how to distinguish these differences, which can leave online students 
feeling neglected. Therefore, his team uses analytics on online student data to create individual learning 
profiles that help instructors understand student learning patterns and potential risk factors associated with a 
student’s particular pattern. Also, his team manages the open-scale courses (a.k.a. at-scale courses) using 
predictive modeling with machine learning. He described to us some of the approaches that ASU EdPlus has 
taken to scale up. 
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Given his role, it is not surprising that Dr. Cunningham considers data one of the most important factors in 
scaling up. He mentioned that EdPlus is taking personalized learning to scale and that this journey involves 
many processes where data is indispensable. He said: 


I think one of the big things is data and, so as we scale up and as tools become available to 
handle bigger and bigger sources of data, that is becoming a real part of the scale equation. 
You're thinking you’re working at scale, but even as we get larger, we want to make our 
learning more personal, so we don't treat everybody exactly the same, but we're targeting 
support and even the course can be changeable according to who the student is. 


Dr. Cunningham described the processes of data collection and data analysis. The process starts with wanting 
to solve a problem or wanting to address a specific research question. For example, when the team wanted to 
know how online students use tutoring services at ASU, such as how, when, and for what purpose students 
were accessing tutoring, they needed to access data that would associate with these questions. The source of 
the data was generated by the tutoring center when students visited the center for help. Dr. Cunningham 
analyzed the data and the EdPlus team leveraged it to make tutoring more effective. Later, data visualization 
specialists could turn that tutoring data into a dashboard where EdPlus executives can see how much their 
tutoring usage is on a daily basis, what kind of courses are accessing tutoring the most, and how that support is 
serving their students. The result is that better decisions can be made about making tutoring available to 
students. 


Throughout the conversation, he reiterated the impact of having partnerships with different corporations in the 
scaling-up process. For instance, ASU has partnered with some of the largest companies in cloud computing to 
standardize the processes and undertake data collection from a wide range of regional issues. Researchers and 
practitioners can then integrate information from individual communities to formulate possible region-wide 
implementation strategies. Also, he said that the partnership sometimes is more accidental. One example he 
shared is the single-platform university vs. the multi-platform university conundrum that ASU has had for 
years, with the take home being that it is important to plan for change. During the development of ASU 
Online, ASU took a multi-platform approach by shifting between cloud computing platforms to meet the 
current needs of the organization. He said, “It's analogous to not committing yourself to only having a PC or 
only MACs with the entitlement of having both because they both have their strengths and their place.” 


He explicitly mentioned that the innovative culture and atmosphere from the inside out makes a positive 
contribution to the fast growth of EdPlus. He told us that he has a lot of freedom in his job to see a problem 
and then try to solve it, resulting in constant innovation at EdPlus. He said, “I think that ASU prides itself in 
being very innovative, and one of the ways that ASU makes that happens is to give people the freedom to try 
new things, and I think that's very powerful.” He used himself as an example. The marketing team has been 
working with outside vendors to create predictive models for whether a student conducts a search online 
relevant to their ‘online education’. The marketing team uses the latest technology available in predictive 
analytics and cloud data to make and analyze those predictions. Cunningham wants to take the same tools and 
predictive models as a template to analyze student learning at scale. For many companies, making this leap 
would not be easy due to the risk and cost evaluations. But at EdPlus, he said the leadership gives him the 
freedom to try new things like that and gives him the resources to be able to do it. He thought this kind of 
support for innovations is critically important for being successful at scale. 


From what has been discussed above, it appears that data and partnerships are unquestionably essential for 
scaling up. It is evident that scaling up requires an understanding of how data is used across different 
departments and how partnerships should be strategically chosen, so as not to restrict development. Dr. 
Cunningham also indicated the power of an innovative culture (i.e., one that provides freedom to test new 
ideas) in an organization when scaling up learning. 


TAKE AWAYS 


Key takeaways 


1. Organizations should understand both how data is used across the organization 
and the opportunities for data usage. 


2. Partnerships should be strategically chosen, so as not to restrict development. 


3. Organizations should utilize the power of an innovative organizational culture 
that supports members to try new ideas. 


2.2.2 NON-PROFIT: CHAN ZUCKERBERG INITIATIVE 


CZI has three focus areas consisting of the science of human health, education, and a variety of justice & 
opportunity initiatives. Their work is accomplished through grant making, investing, policymaking, and 
advocacy to accelerate social progress. CZI’s education work has the goal of helping all children reach their 
full potential through ensuring that they have the knowledge, skills, attitudes and agency that are necessary for 
a successful future. Through their collaboration with educators, families, and students, CZI is developing 
educational tools and methods to improve student learning through a focus on the development of the whole 
person (student or educator). To accomplish this goal, learning science and evidence-based practice are 
combined to help teachers support every student’s individual learning in their context, and to develop 
educators and school leaders. 


An initial focus of CZI Education is to connect research results in practical ways to the challenges of the 
classroom. Partners use applied learning science to help educators solve classroom problems. Also, 
collaborators from education research, development and communities are involved in infrastructure 
development that encourages effective implementation of solutions. 


We interviewed Bror Saxberg, the Vice President for Learning Science at CZI and the former Chief Learning 
Officer at Kaplan, about his experiences in these two different organizations (e.g., CZI is a non-profit 
organization, and Kaplan is a for-profit corporation) regarding how these two organizations have scaled up. 


EXEMPLAR EXAMPLES 


Currently, Saxberg is the head of the learning sciences group within CZI’s education initiative, which aims to 
help K-12 schools, educators, and suppliers make better, context-sensitive, evidence-grounded decisions. These 
decisions involve business (e.g., purchasing, development and investment), teaching and learning, and 
professional development. As part of learning how best to help different schools in different contexts, CZI has 
partnered with Summit Learning on the Summit Learning Program, which is a network of 400 schools across 
the US that is using evidence to improve both academic and non-academic outcomes. Summit and CZI want 
to help these schools benefit from iterative evidence-generating practices grounded in learning sciences. 


At the beginning of the interview, Saxberg pointed out that there is a lack of systematic formative assessment 
to generate evidence about academic and especially non-academic outcomes, and this makes it challenging for 
teachers and principals to make good decisions about education practice, including the use of technology in 
teaching and learning. Better learning for students will not come from adding a simulation, automated 
feedback system, or complex multi-step technology-enhanced support alone. Rather, he favors using the 
evolving theories of human development and the latest empirical research on learning science to make 
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decisions from formative evidence about academic and non-academic outcome improvement. Once identified 
as better for learning, technology is a way to deliver such solutions in ways that are reliable, affordable, widely 
available, and data-rich. Saxberg said, “We need to understand what we want a mind to experience that will 
be better for learning and development, before we wheel in any new technologies”. 


The Summit organization has been focusing on the distribution of a systematic set of learning experiences. 
Specifically, through mentoring and training teachers across hundreds of schools, instructors are set up to use 
these learning experiences (including personalized learning approaches and project-based learning experiences) 
within their different communities and contexts. Teachers often adapt students’ learning experiences to their 
families and their community so that students feel more engaged and motivated to learn. A key is to help those 
teachers understand what the “active ingredients” are within instruction, so that their modifications preserve 
the core practice and feedback elements related to important outcomes. This means professional development, 
too, has to evolve over time. 


As mentioned earlier, the use of technology has to complement what teachers are best able to decide and do: 
it is a mistake to think technology’s role in improving educational environments means removing teachers. In 
other domains (e.g., architecture medicine), the incorporation of technology into professional work frees up 
professionals to spend more time on what they are uniquely best suited to do: work with 
clients/patients/students on what really matters to them. 


Saxberg underscored that such an iteratively improving systems view of teaching and learning is essential to 
finding a scalable solution. This means professional development becomes much more central to teaching. As 
with any expertise, once one way of doing things becomes automatic, it requires well-designed extensive 
practice and feedback to incorporate new ways of working, especially as technology provides new resources to 
help. 


School leaders, too, need their own professional development, to help them understand how to help their 
teachers, other staff, and their students engage in increasingly effective, evidence-grounded approaches to 
academic and non-academic development and support. 


If a principal wants to make a major change to what their teachers are doing, then they have 
to allocate time throughout the year for teachers to have enough practice and feedback. Their 
role is to monitor and make sure they can see the changes and be helpful in continuously 
supporting the motivation and effort of teachers — continuous engagement with the change. 


Through evidence-grounded professional development of school and district leaders, they can be set up to 
support choices for each school’s different context and community that takes advantage of the assets within 
that community, while helping teachers, students, and their families learn and develop around new contexts, 
new skills, and new agency for their own success in life. 


TAKE AWAYS 


Key takeaways 


1. Structured materials should be provided when scaling, while allowing 
instructors to have flexible implementation to reflect local culture. 


2. Technology should facilitate the instructor’s work, but it cannot replace 
instructors. 


3. Evidence-grounded professional development should be provided to both 
instructors and organization leaders. 


2.2.3 INDUSTRY: KAPLAN 


Kaplan has been a worldwide education provider for more than 80 years. The organization’s goal of not only 
providing education but expanding educational access and improving instructional innovation. Kaplan has a 
well-known US-based focus on high-stakes test preparation, but also has global professional training to 
improve employee productivity and help with career advancement through professional licensure, 
certification, and corporate training, as well as an extensive international English-language training effort. 
Kaplan also provides educational services to universities through university pathway programs, international 
student recruitment, university hosting, residential design, and student support services. 


Over the last decade, Kaplan has made progress towards using a “learning engineering” approach in their 
course designs and delivery methods: incorporating learning science and better learning measures in practical 
ways to improve the outcomes of students and solve challenges within each learning organization. They were 
an early assimilator of online education and have led in New Economy Skills Training, which provides 
immersive training for aspiring web developers. 


Our respondent is Bror Saxberg is currently the Vice President of Learning Science at the Chan Zuckerberg 
Initiative. Before joining CZI, Saxberg worked as the global Chief Learning Officer at Kaplan for eight years. 
He worked with the global CEO, Andy Rosen, to transform Kaplan into an effective learning engineering 
organization that used learning sciences and sound evidence about learning in practical ways, such as iterating 
improvements for learning outcomes that were relevant to each different learning organization inside Kaplan. 


EXEMPLAR EXAMPLES 


According to Saxberg, getting Kaplan to change their approach was a multi-step process. First, the 
organization needed to be exposed to new information about what evidence-grounded practices might be able 
to do, and early innovators needed to be supported to try out new approaches, and to talk about them with 
colleagues. Then, a more systematic training effort at scale had to be set up, so that all the instructional 
designers could learn key principles from learning science that they could apply in their own learning 
environment. The training alone was not sufficient: a key consideration in disseminating at scale was to get 
management involved in the language and evidence-based processes for learning improvement. Kaplan’s 
global CEO and CFO agreed to add two additional meetings to the existing quarterly business review meetings 
for all the learning organizations within Kaplan. These two additional meetings were focused entirely on 
discussions with each unit CEO on their learning challenges, measures, investments, outcomes, etc. To be 
ready for these meetings, each unit CEO then engaged with their own learning teams and with Saxberg’s team 
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to understand progress since the last such discussion, using the common Kaplan-wide frameworks and 
language about evidence-grounded learning. 


This had multiple benefits for moving the organization to a more evidence-based approach. For one thing, the 
global CEO and the global CFO were clearly willing to commit two full business weeks of their time each year 
to learning-focused conversations with each general manager. This signaled to the whole organization how 
important evidence-grounded learning improvement was, and no local CEO wanted to appear to know less 
about learning engineering in their unit than the global CEO and CFO. 


He stated that: 


Twice a year, every general manager had to get ready for an hour-long conversation about 
learning with the global CEO and CFO, using common Kaplan-wide learning science 
frameworks and good practices. That repeated practice and feedback then served to embed 
these frameworks and practices in the general managers’ minds, which meant that as they 
went about their normal planning processes, just as they had frameworks for digital 
marketing, technology use, intellectual property, and so forth, they also now had internalized 
evidence-based frameworks for learning and evidence-gathering, which led to better plans and 
questions to their own team. 


Another approach piloted within Kaplan involved more deeply unpacking what top performers actually decide 
and do, as a precursor to developing training. Cognitive science research on expertise over several decades 
shows that 1) you have to use data to find the true top performers, because being known as a top performer is a 
different skillset than actually being a top performer, 2) top performers’ minds have so deeply mastered 
decisions and patterns that the top performer him or herself may not be able to fully explain every decision 
their minds make, and so 3) you need a separate, deep interviewing technique to unpack what real top 
performers decide and do, which is often different than what textbook or conventional wisdom focuses on. 


To find top performers, you first have to deeply understand what value top decision-makers’ decisions actually 
bring to the organization: Fewer errors? Longer tenure for clients? Faster accomplishment of projects, within 
time and budget? Quicker closing of sales? Every job category has its own critical value-adds, and these need to 
be made visible and quantified. Once you have identified a job category of high-volume, high-variance, high- 
value decision-makers (e.g., project managers in construction firms, nurses in health care settings, sales people 
in almost any setting, etc.), you can use the determination of value from these decisions to identify, with data, 
a handful of top experts (ideally out of hundreds or more practitioners) whose decisions from which the entire 
organization would get a massive value lift, if replicated across all the practitioners. 


You then do a series of interviews with this relative handful of top performers to unpack how they do their 
work: how do they organize their tasks, what information do they use, what guides them to make one 
decision or another, or use one process over another. These interviews can then be synthesized together — each 
top performer usually only describes a subset of what all their minds are actually doing — and the experts are 
walked through to edit and improve, removing things that only one of the experts uses, or confirming that 
something only one of them mentioned actually is used by all of them. 


Once accomplished, you have a clear outline of the details of decision-making of a top performer. This, then, 
provides a detailed set of outcomes for the development of an evidence-grounded training program, to build 
mastery around making decisions the way a top performer does. 


TAKE AWAYS 


Key takeaways 


1. Organizations should work to substantially increase the understanding (and use) 
of learning sciences in learning and training environments 


2. Organizations should use data (analytics and human system data) to identify the 
characteristics of expertise for top performers 


3. Organizations should engage local and global managers in learning science 
process to improve understanding of the benefits 


2.2.4 INDUSTRY: SQUIRREL AI LEARNING 


Squirrel AI Learning was founded in 2014 by the Yixue Group and is the first K12 educational technology 
company using intelligent adaptive education in China. The company symbol represents “agility, diligence, 
and management” which aligns with the company goal of providing students with real-time and adaptive 
learning systems that cultivate good learning habits over time. Squirrel AI reports the simulated teacher 
provides an individualized learning plan which boasts a 5-to 10-times higher efficiency than traditional 
instruction methods. Squirrel AI offers high-quality after-school coursework in subjects such as Chinese, 
English, math, physics, and chemistry. Students are given a supervised adaptive learning experience that has 
been shown to improve student efficacy in learning and foster engagement. 


Squirrel AI Leaning has more than 2,600 schools and boasts 12,000 instructors in more than 200 cities in 
China. Last year, the company served more than 220,000 students and, overall, has assisted more than two 
million students. Additionally, Squirrel AI has established an artificial intelligence education laboratory and 
has established research partnerships with Carnegie Mellon University and the University of California- 
Berkeley. Currently, Squirrel AI has a research lab called the CMU School of AI Research Lab on 
Personalized Learning at Scale with Carnegie Mellon University. They also have a strategic partnership with 
Stanford Research Institute (SRI) in Menlo Park, California. 


Our respondent, Richard Tong, is a leader in educational technology and serves as the Chief Architect at 
Squirrel AI. He is the Chair of the IEEE Learning Technology Standards Committee and is also the former 
chair of the IEEE Adaptive Instructional Systems (AIS) Interoperability Subgroup. He has worked with 
Knewton, Amplify Education and been heavily involved with IMS Global and A4L (SIFA) and other AI and 
ed-tech community groups in the last 15 years. 


EXEMPLAR EXAMPLES 


According to Tong, the online teaching and learning platform used by Squirrel AI Learning is an Al-driven 
adaptive tutoring system that is used in combination with coaches to provide one-on-one personalized 
learning. As the first and the largest adaptive learning platform in China, they currently have more than 2,600 
learning centers in China. Last year, they had fully paying customers of more than 220,000 students. Overall, 
they have helped more than two million students. 


Tong describes the biggest challenge facing learning organizations in scaling up as the enormous investment in 
money and manpower. As examples, he cited business-to-business (B2B) education organizations, such as 
Knewton and McGraw-Hill. He said that companies such as Pearson and McGraw-Hill had to invest heavily 
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in creating or converting all the content to fit into the adaptive learning model to address students’ personal 
needs in real-time. His organization has invested the last five years and hired more than 200 curriculum 
designers just to build content for the students from first grade to tenth grade in China. 


Tong acknowledged that it is very challenging to deliver such fine-grained content on a large scale and stated 
that there are three pillars to having a real solution that can be deployed in personalized learning at scale. The 
three pillars are data, business operations, and people. As such, data is the “food” for machine learning, and AI 
algorithms directly decide if personalized learning is effective. He emphasized that effectiveness is more 
complex than what can be seen only by the streaming data that is generated continuously by thousands of data 
sources. Other integrated data, which he named the Total Education Experience data (TEE) needs to be 
considered, including emotions, previous proficiency, and learning goals. He compares TEE to a general 
diagnostic test in a hospital: 


Yes, you want to have a specific test to address a particular problem that you're having. But 

you need to know what your vital signs are, your blood cell count is, or what are the other 

factors that could affect this particular disease at this moment. So, you have to have the 

comprehensive data that gives you a post-longitudinal and current state of understanding of 

the student as a whole person. 
Tong pointed out the biggest barrier in product deployment in an adaptive learning organization is the internal 
segmentation between AI algorithms, market, staff training, and evaluation. Critical questions that must be 
addressed are: How can their organization deliver the product? How can coaches and curriculum consultants 
convince the parents and students to participate? How do we consistently maintain quality? Tong elaborated 
that these questions are not only technology problems but are also business operation problems. One solution 
Squirrel AI used in their organization was to have the three pillars under the same leadership so that this 
leader could unite and lead the people with different expertise in an active and concerted effort to promote 
organizational development and harmony. He said: 


If you want to really produce something that is longer lasting at large scale, you have to 

consider all three pillars and then make your organization that can respond to all of those at 

the same time. They should be in a cross-functional team that delivers with a single road map 

with a single agenda. 
He furthered elaborated on two critical components integral to successful organizational change, policies and 
procedures (P&P), and the boundary (i.e., defining the scope of the change). He found that even if people had 
the same command and control, there could still be a lot of misunderstandings and difficulties in pursuing a 
new agenda simply because people tend to focus on what they are most familiar with and what they are used 
to. He said: 


In order to turn vision into executable deployment and the whole solution, we need a lot of 
different resources. We need to be able to make decisions on behalf of other people previously 
you don't have control over and probably don't share the same vision as you do or at least 
don't understand the processes to make those happen, and that is where, I think, a lot of the 
policy and procedures need to be focused on, how to achieve that vision and overcome the 
common risk or obstacles that is preventing that from happening. 


The purpose of P&P is to cut red tape and boost efficiency. Tong emphasized the importance of flexibility and 
agility in P&P: 


P&P is not cut and dry. This requires a lot of ongoing collaboration and innovation. 
Therefore, leadership must have flexibility, and the support and the resources in order to 
make a lot of things happen in a much agile way. 
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Apart from P&P, the boundary is the other factor in how people build an overall organization and how people 
make things happen in a quicker manner. Tong said one thing that they constantly encounter in their practice 
is the availability of data: 


The processes around data, what to protect, how to handle them, and how do we make data available 
in an ethical and responsible way, become the big obstacle because of the overprotection of the data or 
not understanding the different aspects between data and security. 


Tong commented that if organizations want to implement a fully personalized integrated solution, they need 
to pay more attention to the boundary because if there are too many boundaries, they normally don't have 
agility. One goal of P&P is to find a sweet spot in the boundary-agility trade-off. He suggested that ensuring 
transparency in the field would help reduce confusion and misunderstanding. 


In addition, Tong discussed the role of the third pillar in scaling up that includes the people in the 
organization. No matter how important data and operation are, they are all used by people. In particular, 
there are more than 20,000 staff members who are involved in their learning center. Therefore, Tong believed 
that instead of a one-time intensive training, a continuous customized staff training should be used. He said: 


You have to really track them and also use data in their day-to-day life, in this case, just 

looking at the statistics around how they're delivering teaching experience, how they work 

well with learners and so forth. Similar to how we treat learners, continually adjusting, 

monitoring, and providing solutions that can be customized with individual needs. 
He told us that his organization collaborates with different schools, and these schools will inevitably have 
different priorities. One enlightening approach that his organization developed is designing the overall routine 
for staff to make it much easier to implement the best practices. Also, continuous data monitoring and data 
analysis are required by having the conversation going with the assessments of the implementation of the 
training. He said: 


The formal processes normally are ingrained into the daily routines. We are trying to put as 
much as possible of this continuous learning and continuous sharing or training into these 
formal processes, and allocate for that, as part of the process. 


Another interesting point of view he presented is implementing community-based communication. His 
organization has a very diverse audience and diverse connections with collaborators, especially artificial 
intelligence and educational technology experts. One of the best practices they use is to encourage staff to form 
informal groups and attend informal events to share good stories and successful experiences. He uses 
evangelism to metaphorically describe how they are trying to promote this community-based communication. 
He said: 


It's about community. We actually do a lot of work that is volunteer-based and also is very 
informal, very after-work. We would evangelize on how the technology will move, 
evangelize on best practices through standard, through community, through hosting and 
attending events, publishing articles, and producing surveys and interviews. Because when we 
are doing this, we are learning different perspectives and understanding what people are truly 
doing on the ground. 


Overall, Tong provided many valuable insights regarding their scaling up practice. The three pivotal 
ingredients of scaling up shed light on how imperative it is for learning organizations to recognize the 
importance of integrated data, uniting all their forces, and dynamically training staff members in relation to 
building a scalable organization. 


TAKE AWAYS 


Key takeaways 
1. Learning organizations must recognize the importance of integrated data 
2. Stakeholders should be united on the organizations learning agenda 


3. Organizations should provide dynamic training for staff members while scaling 
up the organization 


2.2.5 INTERVIEW THEMES 


Our interviews revealed that experts suggest a “think-before-you-act” mentality when scaling up, as it is 
essential to consider scale from the start of an initiative. Otherwise, opportunity can be overlooked because of 
a single-minded focus on fulfilling the immediate cohort’s needs. 


Technological Infrastructure. This category contains information on the organization’s handling of data and 
content including their protocol for communicating and exchanging data and tailoring content to support 
administrative decision making, staff training, and instructional strategies for teachers. All the interviewees 
kept reiterating the importance and effectiveness of data in student success and staff training. For example, 
Squirrel AI Learning invests heavily in gaining the Total Education Experience (TEE) data and developing 
more reliable personalized learning. 


Training. As its name indicated, this category emphasized the role of the training in the organizational scaling- 
up. It consists of the text in relation to staff training and leadership training. As we develop the technology 
infrastructure, supporting comprehensive human infrastructure is required. All four interviewees mentioned 
that training is an effective way to keep everyone on the same page as the organization scales up. Approaches 
like: 1) customizing staff training with personalized feedback; 2) getting general managers to allocate enough 
time, money, and efforts on training; 3) utilizing workforce data analytics to pinpoint the top performers and 
unpack their strategies in order to incorporate that knowledge into the training program; 4) paying more 
attention to leadership training (e.g., principals and commanders, Global CEO/CFO), and training them to be 
more open-minded, and increasing their understanding of the state-of-the-art in distributed learning 
applications and corresponding innovations. 


Governance. This category highlights the urgent call for cross-sector strategies in organizational governance in 
the future learning ecosystem. Every interviewee mentioned that having responsive leadership is part of the 
scaling up. For instance, Squirrel AI Learning put their three pillars under the same leadership so that they can 
unite and lead the people with different expertise in an active and concerted effort to promote organizational 
development and harmony. Kaplan added two additional meetings onto the annual calendar, devoting two full 
business weeks for learning-focused conversations with each general manager. CZI focuses on helping 
principals expand their expertise about learning and development science, to help them select effective 
professional development for teachers, and iterate on the implementation progress of ongoing practice and 
formative feedback. 
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3.0 HOW COULD AN EXEMPLAR LEARNING ORGANIZATION BE CREATED? 


3.1 OVERVIEW 


The following section provides an example of the scale-up process at a fictional corporate training 
organization. The example provides an application of how an organization could scale up implementing the 
best practices identified above and in our interviews. 


3.2 CASE EXAMPLE 


This exemplar scenario spans the years 2016-2021 for CSP MedTech, a fictional medical software company. 
The scenario demonstrates the scale up of their corporate training over time using some of the identified best 
practices to meet the needs of the company. The company was able to scale up easily because they initially 
implemented a scalable platform. This allowed them to expand when unexpected scaling was needed. 


3.2.1 BACKGROUND 


CSP MedTech began as a medical-office software company in 2008 in Chicago, Illinois. The founders -- 
Jeremy Carter, Mark Speaker, and Hannah Parker -- have backgrounds in software engineering, medical office 
management, and nursing, respectively. The 50-employee operation in 2008 expanded to 2,500 employees in 
2016 with offices in Chicago and Phoenix. An office in Austin, Texas was set to open in March 2020, with the 
employee count expected to increase to roughly 4,000. The first office outside of the United States was 
projected to open in Canada sometime in 2022, which would at least double the number of employees to 
8,000. This office would expand CSP’s offerings to include medical software for hospital use and would bring 
them a new name: CSP MedTech International. 


Up to January 2016, CSP’s training department conducted training in a traditional manner: PowerPoint 
presentations or software/coding demonstrations given by in-house trainers or unit leads. All employees -- 
from software engineers to project managers to UI designers to tech-support personnel -- were trained in this 
manner. The newly hired Training Department head, Suzanne Loder, realized traditional delivery of training 
was not sustainable, especially in anticipation of the Austin and Canada offices opening. Jeremy Carter, now 
CEO, agreed and directed the CFO to allocate funds for Loder to explore online training solutions. 


Loder and her team did extensive research into distributed learning options, noted best practices, then devised 
the following timeline: 


AVE 2019- 
VAURT) PACyA0) 


PHASE 1 PHASE 2 PHASE 3 PHASE 4 


Traditional Environment Blended Learning Fully Online Learning At-Scale 
Assessment and Environment and Environment and Implementation and 
Infrastructure Additional Infrastructure At-Scale Development Thinking Ahead 


Development Development 
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Phase 1: Assess the current and anticipated training needs, take inventory of the technology 
infrastructure needed to support online training, and start the process to, as Loder coined, “get 
ready to innovate.” (2016) 


Phase 2: Begin, test and finish the move into a blended learning environment and build additional 
infrastructure to support Phase 3. (2017-2018) 


Phase 3: Move into a fully online learning environment and study the past phases and future needs to 
expand. (2019-2020) 
Added because of the acquisition of a competitor in mid-2020 


Phase 4: Scale the Phase 3 learning environment to handle the acquired company’s employees, in 
addition to the projected employees in Canada and any in the future. (2021) 


3.2.2 Company growth and training scaling up 


PHASE 1 (2016): TRADITIONAL ENVIRONMENT ASSESSMENT AND IN 
DEVELOPMENT 


RASTRUCTURE 


During the exploratory research performed by CSP MedTech’s Training Department, it was evident that two 
crucial topics had to be addressed: assessing the current training processes and looking at what type of 
technology infrastructure was necessary to support current and projected training needs. 


Suzanne Loder, the head of training, promptly hired an IT specialist and an instructional designer. Both new 
hires had experience moving large businesses from a traditional training environment to a fully online one. 
The instructional designer, Amber Jennings, would assist with the current training materials and what 
processes were needed to move that material and any new material into an online environment. The IT 
specialist, Joe Segura, would work along with select CSP software engineers to assess their current 
infrastructure and build out what was needed to handle the move to online training. 


Jennings agreed that Loder’s initial three-phase plan was appropriate. She took the Training Department’s 
research and walked Loder and the training team through additional research that supported best practices. 
While Loder was aware of the technology and materials needs and the need to start moving materials online, 
Jennings brought up that distributed learning environments need to address the human aspect as well. It was 
imperative to have full technology and human frameworks for trainee social support. Online learning could 
make a trainee feel isolated, and this had been noted as an issue within online learning environments (Ludwig- 
Hardman & Dunlap, 2003). Loder knew this would mean having additional technology resources and learning 
specialists on-staff in order for the distributed learning environment to be successful and scalable. 


During the technology and instructional design assessments, Loder and her team also reviewed their current 
materials and forecasted the needs to handle an increase of nearly double the employees. Jennings weighed in, 
stating that the forecast needed to extend past the Austin and Canada offices. Since the company was 
broadening their software offerings’ audience, this likely would lead to even more additional employees if 
successful. Jennings stressed that the training would need to be flexible as far as materials creation and the 
instructional design personnel would need to grow as well to ensure there are enough learning specialists to 
support the need. Segura added there must be flexibility in the technology as well to support larger training 
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sessions. While their training sessions were relatively small now, as the company grew, the technology had to 
grow with it. 


PHASE 2 (2017-2018): BLENDED LEARNING ENVIRONMENT AND ADDITIONAL 
INFRASTRUCTURE DEVELOPMENT 


During the first several months of 2017, Jennings and Loder reviewed every piece of material that was 
included in the various training sessions. This included training for: 


e New employee orientation (human resources) 
e New software developers (modified AGILE process used by CSP) 
e Current software developers (continuing education, version training) 


During that review, it was determined that the materials for new and current software developers would 
continue to be presented face-to-face, as they required a lead developer typing out code on a lectern computer 
and employees following on their laptops. If the developers had questions, the lead developer reviewed their 
code and suggested changes. Jennings explained that automating this was possible but would require 
considerable work, including having a system that could “auto-grade” the code and return suggestions for 
improvement. 


Jennings suggested, and Loder agreed, that the one area that could be the “test” for a blended learning 
environment would be the new employee orientation. Most of those materials could be presented online in 
video format, and the forms needing employee signatures would still be processed face-to-face. 


Jennings, along with her newly hired instructional design assistants, started on the new employee orientation 
project immediately by first making sure the materials were following standard learning principles and 
strategies (Craig & Douglas, 2019). Jennings and Loder toyed with the idea of competency-based learning, but 
they decided against it, as it didn’t fit the context of their training. 


Research showed that for any learning environment, on-ground or online, the materials had to provide 
appropriate guidance, such as scaffolding (Kim & Lim, 2019) and feedback (Alharbi, 2017). Feedback in an 
online learning environment is not only feedback from the trainer, but also within the materials themselves, 
such as in-time feedback in quizzes. 


In the meantime, Joe Segura, the IT specialist, worked on building the technology infrastructure to encompass 
the needs for blended learning. This included supporting video, as Jennings stressed that video would be used 
to present procedural interactions, such as how to navigate the human resources system to enter hours worked. 
In the instance of the new employee orientation, videos would not be a “talking head with bullet points”. 
Jennings wanted to use video not as an in-person lecture replacement but as a way to model behavior. 


Segura knew that eventually the learning environment would move from the soon-to-be-blended to a fully 
online environment, so he researched the infrastructure needed to grow the technology at a similar rate as the 
growth of the online offerings. 


By the end of 2017, Jennings and her assistants had completed a draft of the new employee orientation 
blended course. The material creation followed accepted instructional design practices, in addition to 
designing the materials with an eye toward research-based effective design and user experience (UX) 
principles. The videos created to demonstrate the human resources and employee systems were comprehensive 
yet still respected human cognitive processing, which can help learners better cope with new material (De 
Koning, Hoogerheide, & Boucheix, 2018). 


24 


Before formally presenting the videos and other materials to Loder, Jennings needed to perform in-house 
testing first, using UX/human-centered evaluation techniques, as it was important to understand the needs of 
the user groups (Giattino & Stafford, 2019). She employed the help of one of Loder’s human resource 
specialists and one of the lead software developers to perform cognitive walkthroughs (Nielsen, 1993, p. 155) 
on the course, with Jennings and Segura observing each. 


In general, the walkthroughs were a success. The technology supported the videos, and the two participants 
were engaged by the interactive features, such as the section quizzes. Overall, both participants gave the “new” 
new employee orientation format a thumbs-up. However, Jennings knew there was more work to do. During 
her and Segura’s observations, they noted quality issues with a few of the videos and, more concerning, delays 
in loading the in-time quiz feedback. Delays meant user frustration, and this could affect the trainees’ trust in 
the training system. The same was true for the stakeholders; if word got to upper management that the training 
had problems, they might reconsider financial and resource support for the remaining phases. Trust and 
credibility from management to employees was imperative. Segura reconsidered the system set-up and was 
able to fix the delay issues within two months’ time. 


February 2018 brought a batch of 10 new employees to the Chicago office. Loder and Jennings agreed this 
would be a good opportunity to perform a task-based usability test of the new employee orientation materials. 
While the in-house testing, such as cognitive walkthroughs, gave them good insights, Jennings had learned 
over the years that they would have little chance of finding users’ needs without testing with actual users 
(Nielsen, 1993, p. 224). 


Jennings incorporated the new employee orientation materials into a task-based usability test, where she and 
her team observed the employees moving through the materials. Preliminary analysis of the data leaned 
toward the positive, but a thorough analysis was necessary before real data-driven changes could be made. 
With user testing data in hand, Jennings and her assistants spent the rest of 2018 moving the new employee 
orientation materials to a fully online state, as well as bringing the additional training materials for the 
software developers into the online environment. 


Segura was planning his own version of user data by way of exploring xAPI, a popular data standardization 
method used to capture performance data within the online environment. This would tie into measuring the 
interaction between trainees in the online environment and even outside of the training environment (Murphy, 
Hannigan, Hruska, Medford, & Diaz, 2016). He suggested x API to Jennings, who said she’d seen it in 
operation and thought it was a good choice. Segura’s plan was to have xAPI up and running before they 
started Phase 3. In addition, he implemented Adobe Sign, which allows for secure e-signature capture, 
tracking and management. This was the last puzzle piece to bring the new employee orientation materials fully 
online. 

PHASE 3 (2019-2020): FULLY ONLINE LEARNING ENVIRONMENT AND AT-SCALE 
DEVELOPMENT 


Growth was modest in 2019, which didn’t provide much ability to test the system at capacity. However, the 
materials for the software developers were implemented fully online by September 2019. These were more of a 
challenge, as there needed to be more than just a video of a lead developer walking through a new technique or 
process. There needed to be the ability to create a community of inquiry (Col), which consisted of fostering 
cognitive presence, teaching presence and -- possibly the most important -- social presence (Garrison, 
Anderson, & Archer, 2000). One proven strategy for learner motivation, especially when training on a larger 
scale, was focusing on interaction, between trainer and trainee, and also between trainees. The social aspect of 
learning online was crucial, and that included interaction between learners, instructors, content and 
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technology (Anderson, 2003). An integrated institutional support system focused on interactions had to be in 
place before the software engineer training materials were completed. 


Loder, Jennings, and Segura brainstormed ideas to add a social component to the software engineering 
training. Social media was a possibility, as this was a way to improve interactions and engagement for online 
learning, especially as class sizes increase (Bingham & Conner, 2015). After doing research on options, Segura 
suggested employing an enterprise-level version of Slack, a comprehensive chat program that allows for 
sharing of information, files, and more. Most of the individual software teams were already using it, so it 
would be familiar to the employees. 


Another challenge they faced was that with a fully online training environment, the Training Department did 
not know when or how software engineers interacted with materials. xAPI helped with analytics, but those 
would be after an employee had started and/or completed the training. Jennings wanted to make sure that the 
individual aspect of online learning was supported as much as the social. This included support for self- 
regulation, such as providing an in-system dashboard for each trainee to view progress, which can be 
predictive of behavior and goal attainment at larger class sizes (Kizilcec, Perez-Sanagustin, & Maldonado, 
2017). 


The addition of social and self-regulation components to the system, both technology-wise and training-wise, 
took the remainder of 2019 and the first two quarters of 2020. During that time, in March 2020, the Austin 
office opened. All materials were delivered fully online and overall, the system was working as they had 
expected. Loder, Jennings, and Segura were able to take a collective breather, knowing their collaborative 
efforts had paid off. That breather didn’t last for long. 


In June 2020, upper management announced that CSP was acquiring a competitor’s business, Kinsey 
Technologies, which was based in Atlanta. The competitor was in its heyday in the early 2010s, but due to 
mismanagement and a lack of resources to upgrade their hospital-use software in a timely manner, the 
company was at the point of filing for bankruptcy. CSP was able to acquire the company for a reasonable 
price. 


CSP’s upper management was excited about the acquisition, as were the developers. They would now have 
access to proprietary medical software for hospitals, which needed extensive upgrading but gave CSP a head 
start to expand into the hospital-use software sooner than expected. 


However, Loder, Jennings, and Segura had mixed feelings. The acquisition meant they would be responsible 
for training 8,000 of Kinsey’s employees who were staying on. Once the Austin office was at full capacity, that 
would bring CSP’s total workforce to 12,000. Loder projected that by 2022, CSP could grow to 15,000-20,000. 
The three of them and their groups had much work ahead of them, as the acquisition would be complete by 
January 2021. 


Segura knew the current technology infrastructure would not support the traffic needed to have fully online 
training for four offices and 12,000 employees. He spent the remainder of 2020 researching the possibility of an 
at-scale environment, which added a Phase 4 to their timeline. 


Jennings jumped right into researching how not just the materials could grow with the technology, but also the 
resources and support. She knew that in order for any growth of this nature to be successful, there had to be a 
solid support structure in place that encompassed instructional designers, training staff and trainees, with an 
emphasis on technology support (Ricci, 2002). 
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Loder did her part, presenting the need for personnel and technology upgrades to CEO Jeremy Carter and the 
CFO. As she had stated when the company started the move to blended learning, Loder reinforced that the 
resources had to grow with the training needs of a soon-to-be 12,000-employee company. Though money was 
tight due to the acquisition, Loder was granted a modest budget to hire new trainers, and additional 
instructional design and IT personnel. 


By September 2020, Loder had a total of six training specialists: two each in the Chicago and Phoenix offices, 
and two being trained to move to the new Austin office. Two additional instructional designers and one IT 
specialist were hired, giving Jennings four instructional designers and Segura three IT specialists. Considering 
the extensive task ahead it was a skeleton crew, but they were determined and confident. 


PHASE 4 (2021): AT-SCALE IMPLEMENTATION AND THINKING AHEAD 


The move to the at-scale environment was more of a marathon, rather than a sprint. Loder, Jennings and 
Segura intentionally made the process slow and methodical, despite upper management’s initial push to “just 
get it done”. The training team was able to set a meeting with the CEO Jeremy Carter, in which they 
explained the process of testing and evaluation and the potential benefits to the training. Having the research 
and UX testing results helped them support their argument. During this meeting, Carter asked them many 
questions about their process and even made a few good points on the learning implementation from a recent 
session on learning and training at a business management conference he had recently attended. The group left 
the meeting with full commitment from the upper management, which provided them the support they needed 
to implement their training and evaluation plan. With management support, Loder was able to expand her 
team to include a learning engineer who could serve as the bridge between IT and training and had specific 
expertise needed for setting up learning strategies within the scaled-up environments. Loder was also able to 
recruit a dedicated UX evaluator to conduct the testing. 


With the Kinsey acquisition complete, it was time to test the modified new employee orientation for those 
employees. Approximately 8,000 employees would be going through the orientation over a period of two 
weeks. This would be the true test of what the training, instructional design, and IT teams had been working 
toward for more than six months. As the xAPI analytics started coming in, Loder, Jennings, and Segura were 
cautiously optimistic. Along with the modified materials, Jennings added in a post-training questionnaire as a 
way to gather more user data. 


By the end of the two-week period, Jennings and Segura had enough data to analyze to last them months, but 
they did find one major issue they hadn’t anticipated: how the “mobile mindset” would affect the training. The 
numbers told them approximately 300 of the employees used their mobile devices to view the new employee 
orientation materials. The videos, quizzes, and other features worked as expected on mobile. However, the 
questionnaires provided useful additional data. Of the 300 employees, 55 commented that the videos might be 
better as several shorter videos rather than the current longer versions. 


It made sense to both Jennings and Segura. The expectation of the mobile user is different than a user sitting in 
front of a laptop or desktop screen; immediate, bite-sized information was expected. Jennings did some 
research on the topic and came across a concept called mobile-based microlearning, which delivered smaller 
learning units and short-term activities (Hug, Linder, & Bruck, 2006). Microlearning was a different animal; 
instead of “retrofitting” the current materials, they would need to rethink how to present it for mobile 
consumption. Loder, Jennings, and Segura considered it worth looking into for future training offerings. 
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Another future possibility came from Segura. His continued research into supporting and managing online and 
at-scale learning environments brought to him the idea of using virtual reality, especially in the case of the 
software engineering training. Creating an interactive environment that also requires real-time human 
collaboration would be a challenge. Jennings’ own research had touched on virtual and augmented reality, but 
the trick was to make sure the environments were better used for reusable topics and had to be supported by 
well-established learning methods (Shubeck, Craig, & Hu, 2016). 


With the hard work, research, and collaboration of Loder’s, Jennings’, and Segura’s teams, along with strong 
institutional support, CSP MedTech International established a strong distributed learning environment. The 
flexibility of the technology infrastructure and learner-centered training materials allowed them to move fully 
into the at-scale realm, setting them up for additional growth. 


4.0 CONCLUSION 


This report presented examples of how a learning organization can scale up its training and education to meet 
the meetings of its learners. It is possible to scale up a learning or training organization, but it takes time, 
planning and effort across the whole organization. We first provided our primary data collection from 
interviews. Our interviews revealed that experts suggest a “think-before-you-act” mentality when scaling up, as 
it is essential to consider scale from the start of an initiative. Otherwise, opportunity can be overlooked because 
of a single-minded focus on fulfilling the immediate cohort’s needs. Every interviewee mentioned having a 
responsive leadership is part of the scaling up. All the interviewees kept reiterating the importance and 
effectiveness of data in student success and staff training. This included both automated data from the systems 
such as that provided by xAPI and data on the use and abilities of participants such as would be collected by 
usability and experimental data collection. A final theme was that along with the development of technology 
infrastructure, supporting comprehensive human infrastructure is required. This requires a full spectrum of 
support for students and instructors as well as teams of learning and IT specialists for developing and 
maintaining the learning ecosystems. After this, we presented a concrete example of our identified principles 
for scaling up using a fictional company. 
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APPENDIX A 


INTERVIEW PROTOCOL 


My name is {Interviewer Name}. I am conducting an interview to develop a case study to collect best practices 
for designing, delivering, and managing distributed learning at scale in academia and industry. The purpose of 
this interview and case study is to generate evidence to engage stakeholders in relevant dialogue regarding 
distributed learning at scale. 


Today, I will be conducting the interview. This interview will last anywhere from 30-45 minutes. The 
information in this interview will be used only for research purposes. Is it ok if I record this conversation? 


Overview and Understanding of the Organizations 
1. What does your organization do? 


2. Would you describe the online teaching and learning platforms that you use in your organization, such as 
their sizes and their objectives? 


Current Ecosystems and Scale-Up 

3. Can you explain how your organization scaled up? 

4. What were the most useful organizational changes for scaling up? 

5. How did you support learning as the organization scales up? 

6. Can you relate any other factors that had an impact on your organization as it scaled up? Specifically, can 
you reflect on such things as technology support, instructional strategies, organizational communication, the 
support structures within your organization, the multidisciplinary collaboration, or other factors? 

The Application of Technology 

7. Would you elaborate on how your organization adopts the technologies in all aspects of scaling up? For 
example, how did your organization deal with internal communication, staff training, and evaluation of 
instructors and students? 


Prior Experiences, Lessons Learned, and Future Goals 


8. Can you share any valuable lessons learned during the adoption/creation of a large amount of technology 
during the scaling up of your organization? 


9. Where do you see the direction of your organization going over the next five years? 


APPENDIX B 


Table 1. 


Coding Scheme by theme with interview examples for each code 
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Theme Code Interview examples 


[Is _ __, =-WNA |. we 
Institutional Internal “Simply having an idea of what you want to do, is not the same 
Support Communication as getting it disseminated and implemented. And so there's a 

whole change management approach that you have to kind of 
think through. One example is to get the practitioners themselves 
to talk amongst themselves about why this is great.” 


a 


Staff Training/ “T think it's also happened to the staff who are related to 

Faculty delivering that solution, right? Because, for our case, around our 
Development/ schools, we currently have more than 20,000... not our 
Teachers Support employees but the staff members who are involved in our 


learning center and so forth. For them, a lot of times, it's really 
about how we give them customized training continuously, not 
only to attend intensive... like four weeks of on-premise type of 
training but also come back and be able to still get additional 
support as well as always refreshed training on areas that they 
show either need or where we feel that it's something that they 
need to be more prepared for.” 


ff . °» of ~. ... .— =f = - _ .—-.. ss =. —- = 
Multidisciplinary “Being able to work closely with instructors and course 
Collaboration coordinators and leadership in the math department is super 
important because we want to create tools and do an analytics 
that is aimed at solving real problems, so we want to work very 
closely and coordinate very closely with the math faculty and the 
math department, so we're not fixing a problem that nobody 
has.” 
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Students Personalized Learning ‘The way that we approach personalized learning... The reason 

Support that our lab is called the CMU School of AI Lab on Personalized 
Learning at Scale is actually to attack this problem in a holistic 
way. Because we feel like when we're talking about 
implementing this AI as well as personalized education, it's not 
really just about the algorithm, it's not really just about the 
models and so forth. There's three pillars to having a real solution 
that can be deployed and can be used. The three pillars are that 
we have to have content, we have to have operations on 
pedagogy, and also we have to have the technology. Those three 
things are the key ingredients to have this personalized learning” 


a 
UX Considerations/ “What's really cool for us to work on together is, over years, we'll 
be in a very practical question of, how do you modify learning 
Learning Experiences experiences to help them generate additional evidence about 
Design nonacademic development, along with the evidence they're 
generating about academic development?” 


a in ee ee 
Technology Instructional “For example, Canvas. If you think, whenever a student opens 
Support Strategies up their course in Canvas, they are generating data, and we're 

capturing that data, and we're putting it into a data lake, and 
we're going to use that data to do predictive analytics and to do 
analysis of student behavior to begin help on personal 
instruction.” 
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External Collaborations and “Currently, we have a research lab called the CMU School of AI 
Collaboration Partnerships Across Research Lab on Personalized Learning at Scale with Carnegie 
Sectors Mellon University. Also, we have a strategic partnership with 


SRI, usually called Stanford Research Institute, in Menlo Park, 
California. Those are our largest collaborators. But, we also have 
former collaborations with UC Berkeley, with University of 
Memphis, and also NSF Center for Big Learning at University of 
Florida, and several other ongoing collaborations that are in the 
personalized learning research basically.” 


i .—_ =. —,- — 1. _ —_ —_ =—. —— o-oo 
Community-based “It's about community. We actually do a lot of work that are 
Communication volunteer-based and also is very informal, very after-work even 

though it's also part of my work, but we are trying to promote 
this, I would say, evangelism, evangelize on how the technology 
will move, evangelize on best practices through standard, 
through community, through a lot of these events, and 
publishing articles, doing the work like with you guys, produce 
maybe in these type of surveys, and help out with these informal 
learning opportunities.” 


